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TEACHER VARIATION IN PROGRAM IMPLEMENTATION 
AND STUDENT ACHIEVEMENT OF PROGRAM OBJECTIVES 

Robert J. Hess '. • • 

Alberta M. Rogers 

During the recfent past, the emergence of curr'ic!ula and inter- 
vention programs developed by federally funded laboratories and 
R&D centers has resulted in a wide range of progr;ams 'and materials 
designed to foster student learning in diverse areas. Simultan- 
eously, there has also been an increased focus of attention on the • 
^ conduct of educational program evaluations. In many program eval- 
uations, the focus of concern is on the .determination of the extent 
•to whxch specific student outcomes purported to be effected by the 
installation and use of particular programs and materials indeed 
occur. 

When the primary focus of evaluation is. concerned with assess- 
ment of student outcomes relative to program objectives, the inter- 
pretations of outcome data appear to be relatively simple. That 
is, the evaluator can state that the program components were in- 
stalled in a certain number of classrooms, the teachers were pro- 
vided necessary training prio7: to usage of the program components, 
and students did or did not achieve at a level high enough to war- 
rant continued usage. .When provided with this type of information, 
one is tempted to draw conclusions about the mevit of the program. 
This type ot evaluation strategy, however, may lead one to draw 
erroneous conclusions since one assumes stability of treatment with- 
negligible variation between teachers in their adherance to speci- 
fied guidelines for program usage. 

Few, if any, curriculum programs are totally prescriptive. 
Teacher variation in program implementation is likely to occur. 
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Specifying materials and usage procedures in either training com- 
ponents or teachers* guides does not guarantee that teachers imple- 
ment the program in the manner intended t>y the program developer. 

• The fragile status of new programs has been amply docvunented. 
Although studies of implementation have not generally focused on 
programs with well defined objectives and procedures , accompanying 
training programs^ built-in record keeping systems^ monitoring, and 
other self-renewing feedback components; the available literature 
does make it clear that one is naive to assume effective implement- 
ation and use. The literature on diffusion and implementation has 
not generally been concerned with the effectiveness or worth of a 
program but about whether and how new programs are used in schools. 
The literature indicates, however, that new programs are generally 
not used as the program sponsor or developer would like them to be 
used and often are not implemented at all (Gross, Giaquinta, and 
Bumstein, 1971). 

A review of the literature on in^lementation and diffusion 
reveals some of the problems involved in the effective implement- 
ation of educational innovations. 

a) Values and goals as articulated by users often have 
little direct influence on innovations (Goodland, et. al., 
1970; Smith and Keith, 1971). 

b) ' The process of role change required from users has been 

misunderstood and neglected. Often innovations require 
unlearning and releaming and create uncertainty and 
concern eibout competencies to perfom these new roles 
(Joyce, 1969; Pellegrin, 1973; Wachaster, 1973; Jones, 
1973). 



c) The users of innovations seem to be relatively passive 
adopters of recent innovations. Too often, primacy is 
given to innovations rather than to the user's capacity 
to innovate (Charters, 'et. al., 1973; Miles, 1964). 

d) In many cases innovations are not implemented by the 
user as the developer intended or there is considerable 
variation in implementation (Connelly, 1972; Mahon, 1972; 
Pullan, 1972; Gross, et. al. , 1971; MacKinzie, 1970; Herron, 
1971; Gallagher, 1966; Rosenshine, 1970; Solomon, et. al. , 
1972; Hess, 1974). 

Educational innovations rarely interact directly with students; 
interaction is mediated for better or worse by the teacher or school 
administrator and, in many instances, teachers are highly independ^ 
ent agents with respect to externally developed curricula. 

Surely one necessary element for effective change in our schools 
is good curricula, curricula that has been carefully tested, revised, 
and is usable. Prograins with specific objectives and identifiable 
Outcomes are certainly needed, since not many studies have shown 
that general approaches and ideal conceptions will not, by themselves. 
bring change to schools. Curricula must be properly used, and it is 
the use patterns and varieties of implementation strategies that 
must be studied to determine how to best use the products that have 
been carefully developed at considerable expense. • 

There is a desperate need for information on variables related 
to implementation, both positive and negative. These variables 
may be intrinsic to a specific curriculum program and/or variables 
of a more general conception relating to organizational structures, 
teacher support facilities, and, perhaps ^ teacher personality or 
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cognitive style and structure variables. In the absence of . 
methods and measures for characterizing the degree of program 
implementation, it is difficult to provide a realisti^..-3p]praisal 
of the efficacy of a program basiad soley ^^j&ttlSent outcome data 
(Stake, 1967, p. 5). ^^^^ 

By using appropriate implementation measures, however, the 
evaluator cam systematically look at a program as it is installed 
in various sites and examine the degree to which implementation 
variables are present and are related to any teacher training 
components (if present) and/or student attainment of program 
objectives. Such information has relevance not only for 
suirunative evaluations and for comparative analyses of program 
effectiveness, but also for formative evaluations which provide 
data to program developers for revisions of program components 
needing additional emphasis. 

Obviously, the most economical and, perhaps, easiest way 
to assess program implementation would be to make use of existing 
instruments. For example, if a program has as one of its compon- 
ents the prescription that the teacher teach indirectly, then the 
interaction analysis system of Flsuiders (1968, pp. 257-2c5) would • 
be an appropriate measure of this implementation component. Or, 
if the program prescribes that teachers ask many questions that 
require of students divergent thinking responses, the interaction 
system of Gallagher and Aschner (1968, pp. 219-233) would seem 
appropriate* 

Some curriculum programs, however, do not have a specifica- 
tion of the type of teacher-pupil interaction as a process variable. 
In such situations, new instrumentation must be developed that is 
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program specific to measure the degree of . implementation of that 
program's components. A program may provide materials, trans- 
parencies^ student workbooks and either manipulatives with specific 
suggestions concerning the teacher's. use of these components. In 
such instances, the determinatioh of the degree of implementation 
would focus on teacher losage variaibles and the relation between 
variations in usage and student achievement. In addition to actual 
"usage" variables, certain ^teacher variables" may be considered 
as mediating usage (i.e., teacher affect towards the program, 
amount of teacher preparation for lessons, etc.) and could be 
incorporated in implementation models to more fully characterize 
the degree of implementation. 

The goals of the present study were to specify relevant 
implementation variables in the context of an implementation model, 
develop suitable measures , and test the relationship between the 
implementation variables and student achievement. 

JIETHOD 

During the 1974-75 school year, 108 kindercrarten classes in 
the St. Louis Public School System were involved in the Kinder- 
garten Extended Day (KED) Program. The Kindergarten Extended Day 
program is a Title I program designed to provide supplementary 
instruction to identified Title I eligible students deficient in 
^s^ic skills areas. The KED program extended the regular half- 
day kindergarten session into a full-day session using supple- 
mentary curriculum programs and materials to provide remedial 
instructioii. 

One of the critical copfiponents of the KED curriculum was the 
Language and Thinking Program developed by CEMREL, Inc. and 
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published by Follett Publishing Company. Language and Thinking 
is an instructional program which consists of a series of activities 
packages for the development o^ essential skills in language and 
basic concepts. The complete series of packages is planned to 
provide instructional activities for preschool through primary 
grades • 

The general goals of the LAT program are: 

- te develop visual and auditory awareness and discrimination; 

- to develop the child* s use of the language of the classroom; 

• to develop verbal fluency and increase vocabulary size; 
to develop ordering, association, classification, and 
sequencing skills; and 

- to provide practice in doing critical thinking skills, 
drawing relationships, making inferences, making predic- 
tions, analyzing problem situations, synthesizing ideas, 
recognizing incongruities and analogies, making hypotheses 

* and evaluating situations, events, and actions. 

The program is coinprised of one hundred and twenty-six instructional 
objectives contained in nine activity packages. The Lcinguage and 
Thinking packages contain a teacher's guide, student workbooks, 
manipulatives , audio tapes, transparencies, etc., which are used 
by the teacher and/or students in covering daily lesson content. 
Mastery Learning Criterion Tests are available for each package 
to measure the extent to which the testable objectives of a par- 
-ticular package are being achieved by the students . 

SUBJECTS 

All students frpm designated Title I areas, upon entering 
kindergarten^ are given the Comprehensive Test of Basic Skills 
(CTBS, Level A, Form S, CTB/McGraw-Hill) to determine if they are 
eligible to participate in Title I programs. Students scoring at, 
or below, the fifth stanine for the total of the pre-reading sub- 
tests are eligible for Title I programs. All students particip- 
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ating in the i^ED program met Titl^ I guidelines for eligibility 
i.e., scored at or below the fi-^th stanine for the total of the 
pr^-reading subtests. 

Title I guidelines for the KED program further specified that 
each KED class could have no more than fifteen students. 

INSTRUMENTATION 
MEASURES OF STUDENT ACHIE^TEMENT * 

The dependent variables in this study were student achieve- 
ment pf some of the LAT program objectives as measured by severcxl 
of the LAT Criterion Mastery Tests. It was anticipated that the 
KED classes would cover at least five of the ten LAT packages 
during the 1974-75 school year. These packages were Colors y Shapes , 
Sizes y Directions ^ and Blends . The Colors ; Shapes y and Sizes 
packages are covered by a single criterion test while the Directions 
and Blends packages each have a criterion test. 

The LAT criterion tests are keyed to the measurable objectives 
of the LAT program and, as such, have adequate content validity. 
Reliability coefficients (Kuder- Richardson Formula 20) for each 
of the subtests of Colors-Shapes-Sizes and Blends tests^ are as 
follows: 



These coefficients were obtained from the test results of the 
1972-73 pilot study of the LAT program in the St. Louis Public 
School System. Due to extraneous circumstances, the Directions 
Test was not administered and reliability data is not available, 
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■ Colors - Shapes - Sizes 



Blends 



Subtest 



1. 
2. 
3. 
4. 
5. 
6. 
7. 
8. 
9. 
10. 



.43 
.57 
.49 
. 80 
.72 
.55 
.74 
.62 
.69 
.60 



Siibtest 



1. 
2. 
3. 
4. 
5. 
6. 
8. 



.68 
.23 
.54 
.67 
.52 
.41 
.53 



(N=297) 



(N=294) 



MEASURSS 0? D5GREE OF ICTLEIIENTATION 

The prinary independent variables in th.e study were eight 
variables characterizing facets of implementation of the LAT 
program. A sairrole of the KED classes were periodically visited 
by an observer who rated the teacher on her implementation of 
the lAT program using an. observation form containing the following 



1. Time on Task - The observer recorded the clock time (in 

minutest spent on the LAT lesson 

2. " Teacher Preparedness (for LAT lesson) - The teacher was 

rated on a five point scale measuring the extent to 
which the teacher was prepared to teach the LAT lesson. 

3. Correct Following of Procedures - This scale measures the 

extent to which the teacher followed the procedures for 
a particular lesson as specifiSsd in the teacher's guide. 

4. - Proper Use of. LAT Materials - This scale measures whether 

or not a teacher used particular materials ~ (i.e. , manip- 
ulatives, trcuisparencies , etc.) for a lesson as indicated 
ia the teacher's guide and whether or not the particular 
materials were used according to recommended procedures 
as specified in the teacher's guide. 
5 Teacher Effectiveness in Maintaining Student Attention and 
Elicitation of Student Responses ~ The teacher was rated 
on a five point scale characterizing the extent to which 
the teacher was able to maintain a high level of student 
attention and response during the lesson. 



scales : 
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6. Student Interest (in the lesson) - The teacher was rated 
on a five point scale characterizing the extent to 
which students appeared interested in a lesson, 
enjoyed the activities and volunteered to participate. 

!• Extensity of Coverage (of lesson objectives) - This scale 
^measures the extent to which the teacher actually 
covered the several objectives that should have been 
covered for a particular lesson as specified in the 
teacher's guide. 

8. Reported Preparation Time for Lessons - Immediately after 

each classroom observation ^ the oBserver asked the teacher 
to indicate approximately how much time the teacher 
devoted to preparation for LAT lessons. 

A classroom observation form was completed for each observation. 
A copy of this instrument and instructions for its use are pre- 
sented in Appendix I. 

Only six of the eight variables are strictly implementation 
Vciriables. Teacher. J!ffectiveness and Student Interest are variables 
of a more general conception that could mediate student achieve- 
ment in virtually any curricula area. It is very likely that an 
^•effective" teacher would generate a high level of student interest 
which would certainly effect student achievement even in the 
teacher and- students had no packaged curriculum materials. 

In addition to the data obtained on implementation, data"'^^^ 
one other independent variable was collected in order to dis- 
entangle the effects of students* initial ability level from 
the effects of implementation on the measures of student achieve- 
ment. The most readily available measure of student's initial 
ability was the total Pre-Reading sc^re yielded, from the Compre- 
hensive Test of Basic Skills. Although all students in the KED 
program were at, or below, the fifth stanine, there is consider- 
able room for variation within this eligibility criteria. 
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l>RQgEDURES 

Teacher Inservice 

Immediately prior to the beginning of the 1974-75 school year, 
all participating KED teachers and aides were provided inservice 
training in the use of the LAT, DUSO, and BRL components of the 
KED curriculxam program. Teachers were also given instruction in 
the administration of the Comprehensive Test of Basic Skills. 
During the latter part of November, 1974, KED teachers were 
provided further inservice training in the use and administration 
of the LAT Criterion Mastery Tests. 
Student Testing and Selection 

During the first part of September, 1974 , all kindergarten 
students in designated Title I areas were administered the Comp- 
rehensive Test of Basic Skills. ^udents meeting the Title I 
eligibility criteria were eligible to participate in the KED 
program. Sufficient funds were not available to establish 
enough KED classes to accomodate all of the eligible students. 
Within particular schools, the lowest scoring students were 
selected first for participation. *When existing classes were 
filled, remaining students were placed on a waiting list and 
could enter the program when vacancies occurred. A total of 108 
KED classes were thus formed, each containing 15 students. Students 
in the KED classes attended their regular kindergarten class ' (with 
the regular kindergarten teacher) for one-half day receiving 
instruction according to the Kindergarten Curriculum Guide. For 
the other half-day session, the KED students received instruction 
using the LAT, DUSO, and BRL programs presented by a Title I 



12 



11 

teacher and aide. 

The administration of the TIBS, identification and selection 
of students^ cuid formation of the KED classes was not completed 
\antil the middle part of October^ 1974. 

Sampling of Classes and Scheduling of Observations 

Although 108 KED classes were formed^ a sample of only 30 
classes was obtained for purposes of the study. Classes were 
not selected at ramdom, but were selected from schools repre- 
senting each of the four stibdistricts of the St. Louis Public 
Schools having large numbers of Title I students. Classes were 
also selected from schools that were relatively close to each 
other to facilitate quicker travel between schools by the class-, 
room observers. All 30 classes in the sample used the KED program 
during the afternoon session. 

Each KED teacher was asked to indicate at which time during 
the day she would be using the LAT component. Teachers were 
given free choice in selecting the time period but were asked to 
maintain this schedule throughout the school year. Most of the 
3G teachers selected for the study did maintain their schedule. 

Fortunately^ many of the 30 teachers in the sample chose 
different time periods to use the LAt program. This facil'^tated 
the scheduling of observations which was developed such that each 
observer could, under ideal circumstances, observe three classes 
in one afternoon. With two observers, each of the 30 classes 
could thus be observed during^ one week (5 days). 
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Training of Classroom Observers 

In addition to the investigator^ two Evaluating Assistants of 
the the St. Louis Public Schools visited classrooms and rated 
teachers using the classroom observation rating scale. The 
majority of the classroom observations were conducted by the 
Evaluation Assistants. The observers were trained in the use and 
scoring of the observation rating scale. 

The observers were ins\:ructed not to coir^lete an observation 
rating scale form in the presence of the teacher but to complete 
the form immediately after leaving the classroom. Observers were 
told to briefly note information gfermane to the scales during the 
observation of the class but to attempt to write these notes in an 
outline format so as to not arouse the suspicion of the teacher. 

Observers were also told to disguise the purpose of the visits 
by informing the teacher that the purpose was to merely obtain 
feedback from the teacher concerning the usad^ility of the KED 
con^onents and to determine whether or not students appeared to 
enjoy the materials. Teachers were never told when their class- 
rooms would be visited. 

Prior to the actual collection o.?: the implementation data, the 
investigator accompanied one observer (#1) to several classes to 
monitor the use of the observation form. (The other observer 
could not attend) . Initial problems concerning the use of the 
instrument were worked out cuid observer #1 stibsequently monitered 
the use of the instrument by observer #2 dxiring several joint 
observations • 



Classroom Observations 

Classroom observations began soon after the KED classes were 
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formed. Initially, observations were scheduiled to enable each 
observer to visit each of the 30 classes • Problems were encoxantered 
during the latter part of October when classes were preparing 
decorations for Halloween to the exclusion of using the KED 
curriculum components. Also, the observers could not always 
devote full time to classroom observations due to other respon- 
sibilities. The initial scheduling of visits was abandoned and 
a new schedule was developed in which each observer was responsible 
for a given set of classes. The investigator served as a "roving 
back up" and attempted to cover each of the other observers' 
classes to obtain as many observations as possible. 

No data was collected during the week of the Thanksgiving 
holiday or during the week beginning the Christmas holiday since 
students were busy with classroom decorations. No data was 
collected if a teacher was £ibsent when the observer visited a 
class and a substitute teacher or the teacher's aide was using 
the KED components. 

Circxomstances were never ideal and fewer observations were 
conducted than initially expected. Given the unexpected contin- 
gencies of normal school operations, e.g., field trips, guest 
speakers, teacher absences, school events held in the auditorium, 
room decorating, et J. , the expectation of thirty classroom obser- 
vations per week was unrealistic. Another problem encountered was 
the amount of time required of the observer at each site. Many 
times teachers and principals would have questions about using 
the cirriculum components, supplies not delivered, and general 
school system gossip. In order to maintain good relationships 
with the teachers and principals, observers often had to spend 
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some additional time at each school. These types of situations 
often kept the observers from their next scheduled classroom 
visit. 

In order to obtain reliability data on the implementation 
ratings, the observers would periodically visit classrooms 
together and the data from these joint visits was used to deter- 
mine inter-rater reliability,, In order to obtain a maximum 
number of visits , the data for reliability had to be gathered as 
efficiently as possible. Consequently inter-rater reliaibility 
was determined between the investigator and observer #1 and 
between observer 1 and obser 2. No inter-rater reliability was 
established between the investigator and observer 2. 

RESULTS AND DISCUSSION 
Some Preliminary Considerations 

Of critical inportanc-;: to the study is the validity and 
reliability of the measures of the independent variables i.e, , 
the implementation rating scales. Prior to, and throughout the 
study, the measures of implementation can claim only face valid- 
ity. The question of predictive validity will be answered in 
the subsequent analyses of the data. 

Inter- rater reliability of the measures was determined by 
correlating the ratings yielded from several joint observations. 
These results are suiranarized in Tcible 1. Joint observations 
occured between observers 1 and 2 and between observers 2 and 3. 
The data represents joint observations of different teachers at 
different times during the school year. Thus, joint observations 
were obtained during the use of several LAT packages but the data 
was combined to obtain an overall index of reliability. Percent- 
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ages of total agreement between observers on each of the scales 
is included since it is possible to have 100% agreement between 
observers on the ratings for a given scale and yet obtain a 
correlation coefficient of zero (actually r is undefined) if 
there is no variation between the teachers on a given scale rating. 
Inter-rater reliability coefficients, therefore, must be examined 
in the light of the extent to which raters are in agreement on 
scale ratings. The data presented in Taible 1 indicates that the 
inter-rater relisibility of the impleiiientation measures, although 
less than optimal, does not preclude further amalyses of the data. 

An additional analysis of the data yielded from the- joint 
observations was performed to determine whether or not any 
observers were systematically more lenient or rigorous in their 
ratings of teachers. A t-test (two-»tailed) was performed for 
each of the implementation x^ariables using the mean ratings 
yielded from the joint observations of observers 1 and 2, and 
observers 2 emd 3. None of the t statistics approached signifi- 
cance indicating that there were no significant differences 
between observers • 



TABLE 1 

INTER-RATER RELIABILITY COEFFICIENT AND PERCENTAGES 
OF TOTAL AGREEMENT ON RATINGS OR THE MEASURE OF 

IMPLEMENTATION 

• Observers 1 and 2 Observers 2 and 3 

Joint Observations Joint Observation 
R « 4 N « 8 







Percent 




Percent 


Variable 




Agreement 


X 


Agreement 


Reported Preparation Time 




7b 


.11 


6 2 


Teacher Preparation 


.87 


75 


.57 


75 


Time on Task 


.74 


25 


.67 


62 


Following Procedures 


.98 


50 


.87 


50 


Use of Materials 


1.00 


100 


.33 


62 


Teacher Effectiveness 


.96 


75 


.78 


75 


Student Interest 


.96 


75 


.92 


75 


Extensity of Coverage 


1.00 


100 


.64 


75 
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Colors-Shapes-Sizes Criterion Test 

Although thirty teachers and classes were included in the 
study, data from only twenty-three classes (254 students) was 
available for the analysis of the first criterion test. For the 
seven remaining classes, the teachers either had failed to 
administer the tests, the test results were lost in the mail, or 
in^lementation data was not available. 

The Colors-Shapes-Sizes Criterion Test was administered 
during the middle part of January, 1975, at the end of the first 
semester. 

Students from any of the sample classes who had no CTBS Pre- 
"Reading scores were deleted from the sample prior to the data 
analysis. 

All data was analyzed using Version 5 of the Statistical 
Package for the Social Sciences. A one way analysis of 
variance was performed betw^ien classes on each of the variables 
relevant to the Colors-Shjipes-Sizes package. In addition to the 
Colors-Shapes-Sizes Criterion Test scores, CTBS Pre-Reading data 
was included as was data on each of the implementation variables. 
These results were presented in Table 2 along with the means 
for each class on all variables. 

The results shown in Table 2 reveal significant differences 
between classes on the Colors-Shapes-Sizes Test, CTBS Pre-Reading 
scores, and on each of the in^jlementation variables except 
Reported Preparation (RPREP) . The 'between class differences on 
the implementation variables lend additional support for the 
reliability of these measures since, were they unreliable, the 
scores yielded from the measures would be random and the.r. ■ would 
be no significant differences between classes. 

ERIC 
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TABLE 2 

Clisi Heana and Sumary Statistics of the MAS for 
the IiplenentatioD Variables, and the Colbrs-Shapes-Sltes Criterion teat. 
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X 
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1 
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41.2 


1 * 


18.33 


4.67 


21.67 


- AO 

.83 


AO 

.93 
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5.00 


P AA 

5.00 


AA 

.88 
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11 
11 




29*5 


J 


20.00 


1 AA 

3.00 


01 CA 

21.50 


OA 

.80 


Ca 

.50 


1 AA 

4.00 


1 f>A 

4.50 


4 AA 

1.00 


1 
J 
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1 e An 

15.00 


1 nn 

J. 00 
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15.00 


00 

.28 


1 AA 

1.00 ' 
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3.00 


0 AA 

3.00 


.66 


A 


in 


U 1 




1 


16.67 


3.6/ 


Zl.o7 


.73 


/ 1 

.47 


4.67 


f AA 

5,00 


AO 

.83 


3 


XL 


n 
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1.00 


1 

1 
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• 
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23.00 


e nn 

5.00 


10 nn 

18.00 


.94 


1 An 

1.00 
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5.00 
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5,00 
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1.00 


0 
0 


7 
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18.33 


4.67 
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30.00 
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1.00 
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1.00 
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5.00 


f / 0 

4,67 


4 A A 

1.00 


0 






^ 34 J' 




20.00 


t AA 

5.00 


OA PA 

20.50 


AA 

.90 


A 4 

.91 


1 PA 

4.50 


/ AA 

4,00 


1.00 


in 






Al 1 

43.7 




00 f A 

22.50 


0 AA 

2.00 


At CA 

15.50 


4 A 

.32 


4 AA 

1.00 


H A A 

3.00 


4 ^ A 

1,50 


^0 

.33 


11 


1 c 

1j 




46.7 




15.00 


i f A 

4.50 


1 1 4*A 

17.50 


AO 

.83 


A A 

.90 


# Pa 

4.50 


1* AA 

5.00 


A A 

.90 
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14 


/.7 A 


30.7 




4 P AA 
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1 AA 

1.00 


0 1 AA 

35.00 


.00 


AA 

.00 
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.00 


11 
13 


13 


46.8 


41.0 




n.5o 


5.00 


17.50 


1.00 


1.00 


4,50 


4.50 


1.00 


1A 


5 


46.6 


43.4 




11.50 


2.50 


A A f> A 

22.50 


A A 

.33 


.50 


A AA 

2,00 


A • A 

2.50 


.50 


15 


8 


45.6 


39.0 




15.00 


5.00 


16.67 


1.00 


1.00 


4.67 


4.67 


1.00 


16 


11 


45.4 


37.9 




27.50 


4.00 


30.00 


.75 


1.00 


4.00 


4.00 


.75 


17 


10 


44.9 


14.2 




50.00 


3.67 


38.50 


.91 ' 


1.00 


4.00 


4.50 


.66 


18 


■8 


44.7 


3L5 




20.00 


4.00 


22.50 


.74 


.75 


4.00 


4.00 


.66 


19 


10 


42.4 


34.1 




15.00 


1.00 


22.00 


.00 


.50 


1.00 


1.00 


.50 


20 


10 


42.4 


51.0 




20,00 


3.67 


19.00 


,74 


.78 


4.00 


4.33 


.89 


21 


9 


40.1 


13.1 




30.00 


3.33 


20.00 


.94 


l.OO 


3.67 


4.00 


1.00 


22 


\k 


38.3 


29.6 




20.00 


2.00 


18.50 


.25 


.37 


4.00 


4.00 


.58 


23 


14- 


36.7 


30.8 


t 


22.50 


5.00 


32.50 


1.00 


1.00 


5.00 


5.00 


.75 


df 




22/229 


22/229 




,22/27 


22/27 


22/27 


21/26 


22/25 


22/27 


22/27 


22/26 • 


F 




13^36*** 


7.85*** 




1.60 


3:99***;2.64** 


4.27** 


2.43* 


3.82** 


7.12**** 2.28* 


Bartlett- 


5.32* 


5.30* 


Cochran's C 


.26 


.19 


.25 


.50* 


.13 


.22 


.43* 



Box F 

*p .05 ' 
**p .001 
***p .0001 
****p ,00001 
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The application of Bartlett's (1937) and Cochran's (1951) 
tests for homogeniety of variance revealed heterogenous variances 
for the Colors-^Shapes-Sizes test, CTBS Pre-Reading scores, 
Beported Preparatior., Use of Materials, and Extensity of Coverage. 
Both of these tests for homogeniety are also sensitive to the 
assumption of normality .i.e. , that the observations in ANOVA are 
san^led from normal populations (Winer, 1962). Violation of the 
assu2i5)tion of normality, however, is of almost no importance and 
the effects of non-normality on the nominal level of significance 
, of the F-test are extremely slight (Glass and Stanley, 1970). 

Heterogeneous variances effect the level of significance such 
that the actual probability of a Type I error may be larger or 
smaller than the specified nominal probability (Box, 1854; Box and 
Anderson, 1955; Scheffe, 1959). When the n's in each cell or 
class are equal, however, the actual probability of a Type I 
error is very close to the nominal probability (Scheffe, 1959; 
Winer, 1962; Glass and Stanley, 1970). The twenty-three classes 
conqprising the sample, however, differed considerably in the 
numbers of students contained in each class. One class had as 
few as five students and other classes had fourteen and fifteen 
students. In summarizing the findings of studies concerned with 
the effects of heterogeneous variances. Glass and Stanley (1970) 
report; 

Im When the san^le sizes are equal, the effect of hetero- 
geneous variance on the level of significance of the F- 
test is negligible. 

2* When the sample sizes and variances are unequal and fewer 
persons are sampled from the populations with larger 
variances, the probability of a type-I error is* greater 
than a« 
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3. When the sampl^e sizes and variances are unequal and greater 
numbers of persons are sampled from the populations with 
larger variances f, the probability cf the type-I error is 
less than a (p. 372), (see also. Glass, Peckham and 
Sanders , 1972) • 

Regardless of the relationship between sample size and 
variance, the effects of heterogeniety on the alpha (a) level of 
F-test tend to be slight, especially when the number of groups or 
samples is relatively large (Glass, et.al., 1972). Given the 
high levels of significctnce of the F ratios yielded from the 
ANOVA's, it is doxibtful that heterogeneous variances would com- 
promise the results except for Use of Materials and Extensity of 
Coverage, Since the actual alpha (a) level of the F-ratio's 
yielded from the ANOVA*s for these two variables is £<.02, the 
significance of the between class differences is not critically 
affected. 

Regression Analysis of the Implementation Data 

Within the general model two sets of variables are operating. 
The CTBS Pre-Reading scores represent an initial ability variable 
operating at the individual student level. The implementation 
variadDles, on the other hand, operate at the teacher or class 
level. In order to use both sets of varicibles, a two step 

analysis was required. The first step required a regression 
analysis of the CTBS Pre-Reading scores on the criterion 
test scores. Within each class, the mean of the student 



residuals was obtained and this mean residual^ representing 
criterion test variance not accounted for by initial level of 
ability p was. used as the dependent variable for the subsequent 
regression analysis using the implementation variables. 

The means, standard deviations and correlations of all 
variables used in the regression analysis of the Colors-Shapes- 
Sizes test are presented in Table 3. 

TABLE 3 

Means r Standardl Deviations and Correlations of All Variables Used in the 



Variable 


Symbol 


CSSR 






TOT 


FPRO 


uMaT 


TEFF §1NT XCqV 


CSS Residual 


CSSR 
















Reported Preparation 


RPREP 


-.09 


.12 












Teacher Preparation 


TPREP 


.22 












Tine on Task 


TOT 


-.14 


.47 


-.01 










Following Procedures 


FPRO 


.14 


.34 


.92 


.04 








Use of Materials 


UMAT 


.10 


.35 


.73 


-.20 


.69 






Teacher Effectiveness 


TEFF 


.19 


.19 


.81 


-.06 


.86 


.57 




Student^ Interest 


SINT 


.15 


.19 


.52 


-.18 


.60 


.33 


.65 


Extensity of Coveraqe 


XCOV 


.26 


.T)3 


.45 


- .15 


.49 


.31 


• .47 -.24 



Mean .23 20.90 3..70 22.1? .69 .81 3.81 3.50 .92 

S*P' Q.,fy7 8. 06 1.29 6,69 .33 .28 1.17 1.56 .55 

Most of the implementation variables show low^ positive corre- 
lations with the CSS residuals with the exception of Reported 
Preparation and Time on Task. The implementation variables tend 
to show moderate to high positive intercorrelations with the 
exception of Reported Preparation and Time on Task. Given the 
high degree of interrelationship among most of the implementation 
variables r regression analyses of these variables on the. CSS 
residuals would tend to yield an artificially high multiple corre- 
lation, A factor analysis was performed (principal con^onents, 
varimax rotated) on the implementation variables: to obtain 
relatively independent sets of implementation variables i.e.r 
factors . 

The varimax rotated factors matrix for the implementation 
variables is presented in Table 4. Three major factors were 
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yielded from ths analysis. The variables loading on Factor 1 
were Teacher Preparation^ Following Procedures, Use of Materials 
and Student Interest. The factor appears to reflect the teachers' 
personal involvement and attention to detail in the implementation 
' process. Factor 2 contains the variable Extensity of Coverage of 
the objectives. Factor 3 contains the variables Reported Prepara- 
tion and Time on Task and appears to reflect the teachers' time 
involvement in implementations. After conversion to st^dard 
scores, the relevant variables were combined yielding the three 
variables used m the regression analysis. 



TABLE 4 

» Vcurixnax Rotated Factor Matrix of the Omplementation 



Variables for the Colors-Shapes-Sizes Criterion Test 





Factor 1 


Factor 2 


Factor 3 




(Personal 


(Coverage) 


(Time 


Variable 


Involvement) 


Involvement) 


Reported Preparation 


.24 


-.11 


.79 


Teacher Preparation 


.74 


.14 


-.08 


Time on Task 


-.13 


.17 


.66 


Following Procedures 


.97 


.11 


.18 


Use of Materials 


.69 


.06 


.06 


Teacher Effectiveness 


.89 


.06 


.02 


Student Interest 


.67 


-.59 


-.03 


Extensity of Coverage 


.39 


.85 


.07 



The results of the regression analysis (hierarchical inclusion) 



are presented in Table 5. The multiple correlation was not signir 
ficantly different from zero (R =.35 ,F3^19=:. 88) indicating that the 
factors of implementation had no relationship to the class mean 
residuals of the Colors-Shapes-Sizes test. 



Due to matrix singularity an inverted matrix could not be obtained. 
Consequently the initial estimates of the communalities were 
obtained fror the maximum off -diagonal elements of the correlation 
matrix. 

^Matrix singularity precluded the derivation of actual factor scores. 
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TABLE 5 

Regression Analysis of the Colors-Shapes-Sizes Mean Residuals 

Using the Implementation Factors 



Variable 


df 


F 


R2(diff)xlOO 


Beta 


Extensity of Coverage 


1/21 


1.45 


6.05 


-.25 


Personal Involvement 


2/20 


1.09 


4.84 


.21 


Time Involvement 


3/19 


.28 


1.28 


-.11 


R = .35 


3/19 


.88 






R^= .12 










Constant = -.34 











Directions Criterion Test 

Of the original thirty teachers and classes, data from only 
fourteen classes (157 students) were available for the ANOVA and 
correlation analysis for the second criterion test. Many classes 
are not represented due to a lack of implementation data resulting 
from teachers * completion of the Directions package sooner than 
anticipated. Although the laT program guidelines suggest use of 
the Directions package for four to five weeks, many teachers had 
completed the package in two to three weeks. A few teachers, how- 
ever, spent as much as five to six weeks completing the lessons. 

The class' means and stmmiary statistics for the univariate 
analyses of variance for the Directions and CTBS Pre-Reading Tests 
and each of the implementation variables are presented in Table 6. 
Significant differences betweven classes were found for the Direc- 
tions Criterion Test, CTBS Pre-Reading subtest, and all of the 
implementation variadDles except Reported Preparation, and Use of 
Materials. The implementation data was obtained from a total of 
twenty-one productive observations. The mean number of productive 
observations per teacher was 1.5 (S.D. = #51). 
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TABLE 6 

Class Means and Summary Statistics of the /NOVAS for the 
Implementation Variables and the Directions Criterion Test 







DIR PRERONG 


H 


RPREP 


TPREP 


TOT 


FPRO 


UMAT 


TEFF 


SINT 


XCOV 


CUSS 


N 


X 


X OBSER. 


X 


X 


X 


X 


X 


X 


X 


X 


1 


11 


41.0 


49.7 


2 


15.0 


3.0 


20.0 


.64 


.66 


4.00 


5.00 


.83 


2 


13 


39.0 


30.8 


1 




3.0 


15.0 


.75 




4.00 


4.00 


1.00 


3 


13 


37.4 


41.2 


2 


10.0 


5.0 


23.0 


.92 


1.00 


5.00 


5.00 


1.00 


4 


9 


36.6 


50.8 


2 


20.0 


5.0 


20.0 


.90 


.83 


5.00 


3.00 


1.00 


5 


12 


36.3 


43.7 


2 


17.5 


4.0 


35.0 


.71 


.66 


5.00 


5.00 


1.00 


6 


11 


36.0 


45.3 


2 


40.0 


5.0 


22.5 


.84 


1.00 


5.00 


5.00 


1.00 


7 


4 


36.0 


37.3 


2 


17.5 


5.0 


25.0 


.88 


1.00 


4.50 


4.50 


1.00 


8 


9 


34.7 


35.9 


1 


15.0 


3.0 


15.0 


.00 




4.00 


5.00 


.33 


9 


10 


34.4 


36.0 


1 


15.0 


3.0 


25.0 


.60 




3.00 


3.0b 


1.00 


10 


12 


33.8 


34.2 


1 


10.0 


5.0 


21.0 






5.00 


4.00 




11 


12 


33.3 


41.2 


1 


15.0 


5.0 


20.0 


1.00 


1.00 


5.00 


5.00 


1.00 


12 


14 


30.5 


29.6 


1 


10.0 


3.0 


13.0 


.42 


1.00 


3.00 


3.00 


.50 


13 


10 


30.2 


18.6 


2 


. 15.0 


5.0 


22.5 


1.00 


.83 


5.00 


5.00 


1.00 


14 


7 


29.4 


39.9 


1 


20.0 


5.0 


20.0 


1.00 




4.00 


5.00 


1.00 


df 




13/133 


13/133 




12/7 


13/7 


13/7 


12/7 


8/7 


13/7 


13/7 


12/7 


F 




5.38**** 


5.05**** 




.92 


4.49* 


3.71* 


9.12*« 


2.41 


9.41** 


8.79** 6.48** 


Bartlett 


2.b7** " 


6.63*** 


Cochrans C 


1.00** 


.60** .37 




, 1.00** X.00«* 1.00»* 



Box F 



*p .05 

**p .001 

***p ,0001 

****p .00001 
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Tests for homogcniety revealed heterogeneous variances for 
all of the variables except Following Procedures. Given the 
relatively high levels of significance of the F ratios^ it is 
unlikely that heterogeniety would adversly affect the results 
except perhaps y for Teacher Preparation and Time on Task. The 
actxial alpha levels of the F ratios for these two -uriables are 
£<.03 and £<-04 respectively. Overall^ the ANOVA's for the 
iB5)leinentation varicdDles again lend support for the reliability 
of the measures. 

As with the Colors-Shapes-Sizes Test^ a regression of the 
Directions scores on the CTBS Pre-Reading scores was performed to 
yield residuals. Class mean residuals were computed to be used 
as the dependent variable in the regression analysis. 

The means 9 standard deviations and correlations of all 
varicdDles used in the regression euialysis for the Directions 
residuals are presented in Table 7. 



TABLE 7 



Mans, Standard Deviations and Correlations of All Variables 

Osed in the Regression Analysis for the Directions Residuals (N«14) 



Variable 


Symbol 


DIR 


RPREP 


TP REP 


TOT 


FPRO 


UMAT 


TEFF SUIT XCOV 


Dir. Jle^idual 


DIR 
















Reported Prepzuration 


RPRBP 


-^03 














Teachsrr Preparation 


TPRBP 


-.35 


.28 












Tim on Task 


TOT 


.08 


.17 


.33 










Following Procedures 


FPRO 


-.12 


.22 


.78 


.36 








Use of Materials 


OMAT 


45 


.11 


.44 


-.46 


.20 






Teaser Effectiveness 


TEFP 


.10 


.26 


.78 


.42 


.57 


-.07 




Student Interest 


SINT 


.10^ 


.35 


.52 


.26 


.33 


-.36 


.74 


Extensity of Coverage 


XCOV 


.12 


.30 


.59 


.57 


.89 


-.09 


.49 .21 


Mean 




-.12 


lb. 




21.21 


.74 






S.D. 




2.91 


7.72 


.97 


5.38 


.28 


• 15 


.74 .75 .22 
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Table 7 reveals that most of the implementation variables show 
moderately high^ positive intercorrelations with the exception of 
Use of Materials which has negative correlations with Time on Task, 
Teacher Effectiveness, Student Interest, and Extensity of Coverage, 
None of the correlations between the implementation variables and 
the class mecin residuals are very high and several are negative. 

Problems of multicolinearity precluded a meamingful regression 

analysis and again required a factor analysis of the . implementation 

variables. Two factors were yielded from the analysis and the 

4 

variables and factor loadings are presented in Table 8. 



Table 8 

Varijaax Rotated Factor Matrix of the Implementation Variables 



for the Directions Criterion Test 





Factor 1 


Factor 2 


Factor 3 




(Personal • 


(Attention to 


(Use of 


Variable 


Involvement) 


Detail) 


Materials) 


Reported Preparation 




.18 


.03 


Teacher Preparation 


.72 


.54 


.42 


Time on Task 


.22 


.51 


-.46 


Following Procedures 


.37 


.82 


.20 


Use of Materials 


-.03 


.04 


.93 


Teacher Effectiveness 


.82 


.35 


-.06 


Student Interest 


.88 


.01 


-.29 


Extensity of Coverage 


.18 


.98 


-.11 



The variables loading on Factor 1 were Reported Preparation^ 



Teacher Preparation, Teacher Effectiveness , and Student Interest. 
The factor appears to reflect the teachers' overall personal involve- 
ment. Factor 2 contains the variables Time on Task, Following 
Procedures and Extensity of Coverage and appears to reflect the 
teachers' attention to detail. Factor 3 contains the variable 
Use of Materials. 



^Due to matrix singularity an inverted matrix could not be obtained. 
.Consequently the initial estimates of the communalities were 
obtained from the maximiim off-diagonal elements of the correlation 
matrix. 
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l^er conversion to standard scor«-?s, the relevant variables were 
coinbined^ yielding the three variables used in ^he regression 
rmalysis. 

The results of the regression analysis (hierarchical inclusion) 
are presented in Table 9. The multiple correlation was not signi- 
ficantly different from zero (R=. 35 ,F2 ,11=. 77) indicating that 
-the combined implementation variables had no relationship to the 
cl2iss mean residuals of the Directions Test.^ 



TABLE 9 

Regression Analysis of the Directions Mean Residuals Using 

the Implementation Factors Derived From the Factor Analysis 

Variable df F R^(diff)xlOO Beta 

Personal Involvement 1/12 lo57 11.58 -.34 

Use of Materials 2/11 .09 .73 -.08 

R = .35 2/11 .77 

r2= .12 

Constant = -.14 



Blends Criterion Test 

Data from only thirteen classes (121 students) was available 
for the ANOVA and regression analysis of the Blends Criterion 
Test. Six teachers failed to sxibmit the Blends test scores and 
no iii5>lementation data was available for five classes. The 
iir5>a«!mentation data that is available is based on a total of 
tventy-three productive observations. The mean nvmiber of produc- 
tive observations per teacher was 1.8 (S.D.=.69). 

The class means and summary statistics for the univariate 
analyses of variance for the Blends and CTBS Pre-Reading Tests 



^Matrix singularity also precluded the derivation of fac tor scores 
thtas requiring simple rather than weighted combinations of relevant 
variables. 

^The toltirance level of Attention to Detail uas insufficient for 
inclusion in the analysis. 



TABLE 10 

Class Means and SuDtoary Statlstlco of the ANOVAS for the 
Inplenentation Variables and the Blends Criterion Test 







BLND 


PRERDNG 


N 


RFREP 


TPREP 


TOT 


FFRO 


IMT 


TEFP 


SINT 


XCOV 


CUSS 


N 


X 


X 


OBSER 


X 


X 


X 


X 


X 


X 


X 


X 


1 


10 


24.8 


35.5 


I 


15.0 


4.00 


14.0 


.50 


.50 


13.00 


4.00 


1.00 


2 


8 


23.5 


31.5 


3 


18.3 


4.67 


18.3 


.68 


.63 


,4.33 


4.67 


.92 


3 


13 


.22.4 


30.8 




25.0 


5.00 


19.0 


.83 


1.00 


5.00 


5.00 


1.00 


4 


10 


22.3 


36.0 




10.0 


4.00 


20.0 


1.00 


1.00 


4.00 


4.00 


1.00 


5 


. B 


22.2 


37.1 




11.7 


4.00 


11.7 


.21 


.50 


4.00 


4,67 


.36 


6 




21.5 


37.2 




12.5 


5.00 


21.5 


.71 


.83 


5.00 


5.00 


.83 


7 


11 


21.1 


40.3 




15.0 


5.00 


20.0 


1.00 


.66 


5.00 


5.00 


1.00 


8 


12 


20.4 


41.2 




15.0 


5.00 


24.0 


.92 


.83 


5.00 


5.00 


.75 


9 


12 


20.0 


43.7 




7.5 


4.00 


18.5 


.83 


.75 


4.00 


4.00 


.50 


10 


10 


18.9 


33.6 




10.0 


3.00 


16.0 


.50 


1.00 


3.00 


4,00 


1.00 


11 


11 


16.4 


37.9 




15.0 


3.00 


25.0 


.73 


1.00 


3.00 


3.00 


.50 


12 


7 


18.0 


39.9 




15.0 


5.00 


20.0 


.62 


1.00 


5.00 


4.00 


1.00 


13 


5 


18.0 


52.0 




17.5 


5.00 


21.5 


.64 


1.00 


5.00 


5,00 


1.00 


df 




12/108 


12/108 




12/10 


12/10 


12/10 12/9 


12/8 


12/10 12/10 12/9 


F 




4,68** 3.61** 




3.12* 


1.93 


1.50 


1.66 


1.61 


2.18 


2.10 


1.18 



Bjrtlftt 1.99* 1.06 

*p .05 
**p .0001 



Cochrane C .48* 
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and the implementation variables are presented in Table 10. 
Significant differences between classes were foiind for the Blends 
and CTBS Pre-Reading tests and for Reported Preparation. Tests 
for homogeniety of variance revealed heterogeneous variances 
for the Blends test and Reported Preparation. Heterogeniety 
would not compromise the results for the Blends test but might 
negate the effects shown for the implementation varialale since 
the actual alpha levels for this variable is only £<.04. 

The lack of between class differences for the implementation 
variables does not necessarily indicate unreliability of the 
measures in this instance since the teachers had probably become 
thoroughly familiar with the teacher's guides and program compo- 
nents. Given the teachers' length of exposure to, and usage of, 
the program, a concommitt^^nt decrease in variation in implement- 
ation might be expected. The means and standard deviations of 
the implementation variables across the Colors-Shapes-Sizes , 
Directions, and Blends packages are presented in Table 11. The 
data in Table 11a is based on all teachers having implementation 
data for any of the respective package:i, thus, some teachers are 
not represented across all packages. Inspection of the standard 
deviations of the implementation variable "acr^^^ 

decreased variation across all packages for all of the variables 
except Use of Materials and Extensity of Coverage. The variation 
in Extensity of Coverage remains fairly stable while the variation 
in Use of Materials, although greater for the Blends package than 
for the Directions package, shows more variance in the Colors- 
Shapes-Sizes package thr;n in the other packages. 



TABLE 11a 



Means and Standard Deviations of Implementation Variables 
Across the Colors-Shapes-Sizes, Directions and Blends Packages 
' for Teachers Having Implementation Data on the Respective Packages. 

. Colors-Shapes-Sizes Directions Blends 
Vari able Mean S.D. Mean S.D. Mean SjV^ 

RiF^irTid Preparation 21.10 10.32 16.92 7.72 14.42 4.44 



Teacher Preparation 



21.10 


10.32 


16.92 


3.84 


1.35 


4.21 


22.12 


7.28 


21.21 


.74 


.34 


.74 


.83 


.30 


.89 


4.00 


1.21 


4.39 


4.06 


1.24 


4.54 


,80 


.28 


.90 



4.36 .75 



Time ou Task 22.12 7.28 21.21 5.38 19.19 3.70 

Following Procedures 
Use of Materials 
Teacher Effectiveness 
Student Interest 
Extensity of Coverage 

N GTeaciiersl . w . 23.. .14. .13 



.71 .22. 

.82 .20 

4.2G .83< 

4.41 .63 

.84 .23 



. TABLE lib 



Means and Standard Deviations of Implementation Variables 
Across the Colors-Shapes-Sizes, Directions and Blends Packages 
for Teachers Having Implementation Data on All Packages. 





Colors- 


-Shapes-Sizes 


Directions 


Blends 


Variable 


Mean 


S.D. 


Mean 


S.D. 


Mean 


S.D. 


Reported Preparation 


17.50 


3.87 


17.20 


2.45 


14.52 


5.73 


Teacher Preparation 


3.73 


1.63 


4.00 


1.00 


4.57 


.54 


Time on Task 


19.09 


2.69 


21.43 


6.90 


19.24 


3.80 


Following Procedures 


.71 


.40 


.71 


.35 


.72 


.26 


Use cf Materials 


.88 


.19 


.86 


.18 


.87 


.19 


Teacher Effectiveness 


3.67 


1.30 


4.28 


.76 


4.57 


.53 


.Student Interest 


3.50 


1.56 


4.57 


.79 


4.52 


.50 


Extensity of Coverage 


.83 


.26 


.90 


.25 


.80 


.27 


N CTeachersl «■ 


7 













The data in Table lib is based on teachers having injalenenta- 
tion data for all three packages. Decreases in variance across 
packages are revealed for Teacher Preparation^ Following Procedures 
Teacher Effectiveness and Student Interest. Variation in Use of 
Materials and Extensity of Coverage, remains relatively stable 
while the variation in Reported Preparation and Time on Task 
3hows erratic trends across packages. The data in Table lib, 
however, may be suspect due to the relatively small sample (N=7 
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teachers) and may yield a somewhat distorted view of the overall 
implementation trends. The evidence does reveal^ however^ a 
tendency for teachers to become more homogeneous in their 
-in^jlementation of the program components. 

Regression Analysis 

The means, standard deviations and correlations of all 
variables in the regression analysis for the Blends residuals are 
presented in Table 12. 

■ ' ^ ' ' ~ 

Cleans, standard Deviations and Correlations of All Variables used 

in the^jscyre asio n Analysis of the Blend s R esiduals (N »13) . 

Variable ' " Symbol bKb^lu RprIp '^^^^ t<>T f-pKb uWi'r 'rfit't- 51N1! JiCov 

Blends 2^3 id. BRESID 

Reported Preparation Wrep ''^^ 

Teacher Preparation TPRep .50 

Tiise on TasJc TOT -. ^9 ,19 .23 

Following Procedures FTRo --^^ .10 -32 ,69 

Use of Materials UMAT --^"^ .14 --05 .62 .35 

Teacher Effectiveness TEPf ^'^ .41 ,36 .39 .15 

Student Interest SINT • .39 -80 -.05 .1^ -.22 .78 

Extensity of Coverage . XCOV * . 3 3 /38 .07 . 29 .30 .26 ,27 

Hean =T75 17773 — — 19.19 . .03 4.ib 4.4:^ 7^ 

S,D. ^ 2/15 4,44 .75 3,70 .22 .19 .83 .63 .23 

Table 12 shows that the majority of intercorrelations among 
the implementation variables are moderate to high positive. Four 
of the implementation variables show negative correlations with 



"Blends fe^ TihLe~moclerately high, negative correla- 

tions between Time on Task and use of Materials with the Blends 
residuals may suggest that students tend to become less enthus- 
iastic about the content and length of daily lessons. This may 
be expected since the Blends package content is mainly a synthesis 
of the content of the preceeding packages. 

The varimax rotated factor matrix for the implementation 
variables is presented in Table 13. Three factors were yielded 
from the analysis. The fo^^^ variables loading most highly on 

5^5 
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•Factor 1 are Teacher Preparation^ Reported Preparation, Teacher 
Effectiveness and Student Interest. This factor appears to reflect 
the degree of personal involvement of the teacher. The variables 
loading most highly on Factor 2* are Time on Task, Following Pro- 
cedures and Use of Materials. This factor appears to reflect 
teachers' attention to detail. Factor 3 contains the varicible 
Extensity of Coverage. 



TABLE 13 



Varimcuc 


Rotated Factor 


Matrix of the 




Implementation Variables for the 


Blends Criterion 


Test 




Factor 1 


Factor 2 


Factor 3 




(Personal 


(Attention to 


(Coverage) 


Variable 


Involvement) 


Detail) 




Reported Preparation 


■ .39 


.JLU 


.38 • 


Teacher Preparation 


.96 


.12 


.22 


Time on Task 


.11 


.99 


-.05 


Following Procedures 


.22 


.63 


.12 


Use of Materials 


-.17 


.64 


.33 


Teacher Effectiveness 


.91 


.30 


.11 


Student Interest 


.86 


-.14 


.15 


Extensity of Coverage 


.19 


.13 


.74 



As with the previous factor analyses, matrix, singularity 



required the use of the highest off-diagonal elements in the 

correlation matrix to obtain the initial commiinality estimates and 

precluded the derivation of actual factor scores. The results of 

7 

the regression analysis are presented in Table 14. The multiple 
correlation \TSiS not significantly different fror. zero •CR=^.42vr2, 10=^ 
1.09). Thus the Blends implementation factors are not significantly 
related to the class mean residuals of the Blends test. The nega- 
tive beta weight obtained for the vactor Attention to Detail may 
indicate that students tend to become bored with the content of 



The tolerance l^vel for Personal Involvement was insufficient for 
inclusion in the regression analysis. 



29 



the Blends package. This probably reflects a relative redun-' 
dancy of content since the Blends package is a synthesis of the 
content of the preceeding packages. 







TABLE 14 


°^ Blends Mean Residuals Using the 
the Implementation Factors Derived From the Parrtnr- ar..,"?r=Se • • 


Variable 


df 


F R2(diff)xl00 Beta 


Attention to Detail 
Coverage 

R-= .42 

R"^* .18 

Constant = ~.. 49 


1/11 
2/10 
2/10 


2.07 15.8 -.44 
.25 2^1 .15 
1.09 



Most of the variance in student achievement across all three 



ciriterion tests; can he attributed to initial differences in ability 
a^ measured b^ the CTB§ Pre-R^e^ding sxibtegt. 
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Some Serendipitous Results 

At the end of the school year participating KED teachers were 
given a questionnaire to elicit feedback on the curriculum compon- 
ents of the KED program. A copy of this questionnaire is presented 
in Appendix II. 

The questionnaire was designed to obtain information on some 
variables that could mediate implementation of the LAT components 
of the KED program. Generally^ these variables may be divided into 
demographic variables (i.e., years of teaching experience and 
presence or absence of a teacher's aide), and "reaction" variables 
(i.e., reactions to the teacher's guides, manipulatives , directions 
for using components, etc.). Questions related to the LAT program 
were scored and these scores were correlated with the overall mean 
scores on the implementation variables for twenty-one teachers for 
whom data was available. For this sample of teachers, the imple- 
mentation data was based on eighty-three productive observations 
across packages. The mecin number of productive observations per 
teacher was 3.95 (S.D.=1.60). The implementation means for all 
but five of the teachers are based on observation data obtained 
across at least two of the three packages. 

Since bivariate normality was not observed for these variables, 
Kendall rank-order correlations were computed and these inter- 
correlations are presented in Table 15. Inspection of Table 15 
reveals that most of the intercorrelations among the implemen- 
tation variables are moderate to high positive and significantly 
different from zero. Exceptions may be noted for Reported 
Preparation and Time on Task. 
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Of the demographic variables, Kindergarten Experience (KEXP) 
shows low, negative correlations with most of the implementation 
-V2u:iables but the^e correlations are not gnif icantly different 
from zero. 

Presence of a teachers Aide (AIDE) shows low positive 
correlations with all of the implementation variadDles and these 
intercorrelatior.s are significantly different from zero for 
Teacher Preparation and Student Interest. Apparently the presence 
of a teachers aide enables teachers to more adequately prepare for 
LAT lessons which ^ in turn, may relate to student interest in the 
lessons. 

Under the heading Reactions to Teachers' Guides (Table 15) — 
are varicUiles characterizing teachers' reactions to the organiza- 
tion (LORG) , structure (LSTR) , and sequence (LSEQ) of LAT lessons, 
clarity of directions (CLAR) , and ease with which directions 
could be followed (EFOL) „ Most of the intercorrelations between 
these variables and the implementation variables are low positive 
but not significantly different from zero. An exception may be 
noted for Clarity of Directions (CLAR) , however, which has 
correlations greater than chance expectations with all of the 
implementation variables except Reported Preparation and Student 
Interest. In addition. Lesson Orgauiization shows a better- 
than-chamce correlation with Time on Task* Thus teachers who 
perceive the lessons as being well organized tend to spend more 
time on the lessons and teachers who perceive the directions as 
being clear and unambiguous tend to manifest a uniformly higher 
degree of implementation. 



EKLC 
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The negative correlation between Clarity of Directions and 
Kindergarten Experience suggests that the more experienced 
teachers tend to view the directions as being unclear or ambiguous. 

Under the heading Reactions to Manipulatives are those 
variables characterizing teachers' reactions to the number (NMAN) 
and Usefulness (UFMN) of the manipulatives , whetii'sr or not the 
msmipulatives were hard or easy to handle (HAND) , cuad were 
interesting (INTR) to the students. Number of Manipulatives 
shows moderate positive correlations (beyond chance) with all of 
the implementation variables except Reported Preparation, Time 
on Taskr adn Use of Materials. The variable Usefulness of Mani- 
pulatives (UFMN) shows very low, negative correlations with most 
of the implementation variedDles but none of these correlations 
are beyond chance. Handleable (HAND) shows moderately low, 
positive correlations with most of the implementation variaibles 
but these correlations are not beyond chance except the correlation 
with Extensity of Coverage. Teachers' perceptions of the students' 
interest (INTR) in the manipulatives also shows moderately low, 
positive correlations with all of the implementation, variaibles 
but these correlations also are not beyond chance except for 
Following Procedures. 

under the heading Reaction to Teacher Training is the 
variable Self -Confidence (CONF) which indicates the extent to 
which teachers were confident of their ability to use the LAT 
program as a result of the LAT inservice training. None of the 
correlations between Self-Conf idence and the implementation 
variables, however, are significantly different from zero. 
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Caution should be exercised in attempting qualitative 
interpretations of correlations between the questionnaire vari- 
ables and the implementation variatbles since the questionnaire 
variables have face validity only and there is no evidence of 
the reliability of the measures. In addition, the correlations 
may be suspect due to the restricted range problem since most 
questionnaire variables were scored on a two or three point 
■Scale* 

Summary Discussion 
Preliminary Considerations and Limitations 

An obvious limitation of the study is that neither teachers 
nor students were randomly sampled or rcindomly assigned to the 
KED classes. In the context of this study, however, the sample 
may be considered somewhat representative of a larger population 
of inner-city kindergarten teachers and Title I eligible students. 

A further limitation is that any effects on student achieve- 
ment resulting from the interaction of the BRL Math, DUSO, or 
3»gular kindergarten curriculum programs with the LAT progreim 
are uncontrolled. It is assxamed that euiy such interaction effects 
are uniform across classes or negligible. 

The relatively few productive observations of the teachers 
(especially for the Directions package) also fosters a cautious 
interpretation of the results of the study. 

Summary 

The results show, however, that there are significant 
differences between classes in student achievement of the LAT 
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program objectives as measured by the first three criterion 
.tests* The results also showed that, except for the Blends 
analysis, classes were significantly different from each other 
on the implementation variables' thus lending additional support 
for the reliability of the implementation measures. Although 
the assumptions of homogeneity of error variance and normality 
were violated for most of the one-way analyses of variance, it 
is unlikely that such violations would negate the significance 
of the between class differences given the relatively low alpha 
levels yielded from the F tests. 

Inspection of the means and standard deviations of the 
implementation variables across packages for seven teachers 
having complete data revealed, that as the school year progressed, 
these teachers spent less actual time in preparation for lessons 
but appeared to be better prepared for the lessons. This may 
be a result of increased experience in using program materials. 
These teachers also become more effective and more homogeneous 
in maintaining student attention and eliciting student responses 
(Teacher Effectiveness) and their students appeared to manifest 
more interest in the lessons (Student Interest) . 

An emalysis of the teacher questionnaire data revealed that 
the presence of a teachers aide was positively related to Teacher 
Preparation and Student Interest, indicating that the presence 
of an aide may enable teachers to more adequately prepare for 
lessons which might subsequently affect student interest in the 
lessons. 
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Teachers who indicated that the directions in the teachers' 
guides were clear and xmambiguous tended to metnifest higher scores 
on most of the implementation variables^ Teachers who indicated 
that the lessons were well organized also tended to spend more 
time on task. 

The negative correlations between years of kindergarten 
experience^ perceived clarity of directions, and preparation 
suggest that the more experienced teachers may perceive some 
ambiguity in the directions and' show less overall preparation 
for the lessons. This may suggest a "resistenca to change" 
phenomenon as noted by Gross, et, al, (1971) especially considering 
that kindergarten experience showed negative correlations 
(though not beyond chance) with most of the implementation 
veuriables • 

Although the questionnaire can clcuLm only face validity and 
most of the reactive variables are measured only on a two or 
three point scale, the data suggest that the teachers' perceptions 
of some facets of the program are related to certain facets of 
iii5)lementation . 

In summary, the results show that teachers did vary in their 
in^jlementation of the LAT program but that the variation in 
implementation was not related to clauss acievement. Most of the 
variance in student achievement was due to variance in initial 
ability. 

It is critical in the evaluation process that variations 
in degree of impJementation be characterized. The use of class- 
room observation with implementation rating scales can provide 
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a methodology for the specification of variations in iroplementa-* 
tion. In the absence of information about variations in imple- 
iDentation, the evaluator/ researcher assumes any or all of the 
following: 

1) the educational program and its discrete elements are a 
configuration of a "stand alone" product and, hence, 
"teacher-proof" or 

2) teacher variation in implementation is negligible in its 
effects on student outcomes, or 

3) is not an important variable or variable set to consider, 
or 

4) teacher variation in implementation is a random vari'able 
whose effects can be minimized by appropriate random 
sampling procedures across a large nxamber of teachers and 
classes. 

The first assumption is certainly questionable since the teacher, 
at least in this study, has been shown to be a powerful mediating 
variable in relation to the usage of a program. Currently there 
is a lack of sufficient evidence to provide blanket support for 
assuii5)tions two and three. The fourth assumption requires a 
considerable amoxint of expenditure and effort in the evaluation 
process, cuid may be impractical for labs, R&D centers and school 
systems engaged in curriculum development amd/or evaluation. 

In specifying variations in implementation and isolating 
critical implementation variables, the evaluator can provide 
feedback to the developer or inservice centers concerning whiclx 
facets of the in?>lementation of a program need special emphasis or 

46 
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require modification to enhance optimal usage emd achieve 
maximxim impact. 

Suggestions for Further Research 

The critical element in research on the effects of teacher 
variation in implementation is obtaining suitable measures of 
implementation. Within this context, the major problem is that 
of developing measures applicaible to a variety of curriculum 
programs to enhancti useability. Although it can be argued that 
curriculum programs represent unique .configurations of lessons 
and materials f teacher usage of such programs ought to be 
charaicterized by variables that trsinscend individual programs. 
The teacher implementation variables used in the present study 
may be considered to have wide applicability (for programs having 
at least some structure) but some of the measures seem to lack 
precision. For example, Teacher Preparation , Teacher Effective- 
ness, cind Student Interest are imprecise even though most of the 
scale points for each are defined. The imprecision of these 
variables lies in their treatment as unitary variables having 
no specifically defined sub-components. For example, Teacher 
Preparation could acquire more precision by defining it in terms 
of stib-categories like availability of materials, frequency of 
referring to teacher's guide, and fluidity of transition from 
topic to topic. Each sub-component could be measured by a rating 
scale or frequency count (for number of referrals to the teacher's 
guide) and the summation of the component scale scores would by 
the score for Teacher Preparation. A variable like Teacher Effec- 
tiveness may acquire more precision by actually noting the number 
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or percentage of students exhibiting attenuation responses 

and offering (via hand-raising) to respond to respond to pertinent 

questions or volunteering to participate in activities. 

Although the variable Following Procedures was specifically 
defined as the percentage of specified procedures actually followed 
by a teacher in her presentation of lesson content^ more precision 
could be obtained by focusing on discrete teacher behaviors 
suggested by'' the procedures to be followed in the presentation of 
the lessons. To this end, the work of Bellack etv al- (1966), 
Smith and Meux (1962)/ and Hudgins (1971) concerned with teacher 
"moves" would be geintiane. Further specification of teacher be- 
haviors could accomplished by considering the results of the ' 
teacher verb-sorting studies by Miller et. al# (1967) , Johnson 
(1969) r and Wiley (1969) which demonstrated that from fifty to 
eighty verbs could characterize teacher classroom activity 
in the context of facilitating student learning. The lessons of 
more structured curriculum progreiins could be examined to deter- 
mine which teacher verbs most characterize the procedures to be 
followed (e.g.^ demonstrate, display r illustrate, lecture, etc*) 
and the score for Following Procedures would be based on teacher 
interaction with program conqponents relative to the relevemt 
teacher-behavior verbs. Deviations from recommended procedures 
could be readily characterized by noting which other verbs reflect 
such deviations and this data could be sued to determine which 
alternate behaviors, if anyr have significant effects. (Assxaming 
that Following Procedures itself has a significant effect) . 
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A further line of research should focus on variables that 
mediate reacher implementation. Such variadDles may be classified 
under general headings of preparation variables (e.g., quality of 
inservice training), setting variables (e.g., availability of sup- 
port searvices like curriculum specialists, teacher aides, in- 
service trainers, administrators' interest in the program, etc.), 
and other teacher characteristics (especially personality variables 
and self-confidence) . Such research would be beneficial in the 
development of general models of implementation encompassing 
a wide range of variables that could mediate student achievement 
of curriculum and program objectives. Such studies would 
obviously require large numbers of schools and other resources, 
hut the rigors of scientific inquiry and the obligation to pro- 
vide quality education demands that such research be undertaJf.>>^^ 
to adequately specify the effects of the implementation of in- 
novative curriculum programs. 

Although the implicit assumption is that high levels of 
teacher implementation of a program should result in high levels 
of studetn achievement of a program's objectives, rigorous 
implementation of a relatively structured program could result in 
students being treated as "Pawns" rather than "Origins". Since 
the "Origin-Pawn" variable is related to academic achievement 
(deCharms, 1976) and the relative amount of freedom in a situation 
may induce more Origin or more Pawn feelings (deCharms, 1972), 
forcing students and teachers to act in a predetermined manner 
(e.g., high levels of teacher implementation) may in fact serve 
to hinder student acquisition of program objectives. 
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Such a relationship between the level of prograun structure ^ 
teacher implementation^, and student achievement, if it exists, has 
serious implications for curriculum developers and may demand a 

reconceptualization of the role of curriculum programs in the 

i 

teaching-learning process. 
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APPENDIX II 

LAT CLASSROOM OBSERVATION FORM AND INSTRUCTIONS 



lAT. LEVEL A 
Observation Form 



Date of Observation 

Teacher . 

School ' 

Aide 

Name of LA T 

Title of lAT Lesso n 

Objectives for this lesson (indicate objective number(s) 

*1« Time on lesson 

a. Time lesson begins 

b. Time lesson ends Total time (a-b)^ 

Teacher Impiementafion 

2. T Teacher prepardness for LAT lesson 

1 2 3 4 5 

3* Correct following of procedures as specified in the teacher's guide. 

Actual Number of Procedures Followed 

Total Number of Procedures Specified for the Lesson " 
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4. Proper use of LAT materials 

Suggested Used (1) Used Properly(l) 

-Moterials for Lesson (%/) Not Used (0) Used Improperiy(O) 

Let's Start 

Teacher's Objects Box 
Simple Objects Transparencies 
Lot's Start Audio Tape 
Colofs 

Student Activity Book 

Colors Audio Tape 

Colors Objects Box Items ' 

1) Objects of different colors 

-2) Color blocks ZZZZIZZ IZ^ZZZIIZ 

3) Food coloring & containers 

Color Picture Cards of Real 

Objects 

Clearly Colors Transparency 

Correct & Incorrect Color Cards 

Shopes 

Student Activity Book 

Shapes Audio Tape 

Plastic Shapes 

Shapes Picture Cards A 

Shapes Kcture Cords B 

Shapes-Colors Transparency 

Clearly Shapes Transparency 

Shapes of Things Picture Cards 
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Materials . ^T^^^^ '^'^^^^^ Used Properly(l) 

" for lesson (^/) Nof Used(O) Used ImproperJy(O) 

Blencfe 

Teacliers Guide 

Activity Book for Children ~ 

Color & Shape Bmgo Game " ^ ~ ' 

Geometric Shapes Box 

Audio Tcpe - let's Talk ~ 
Actions 

Action Picture Cards 

Action Transparencies " ' ' 

Animal Action Picture Cords ~ 
Action Audio Tope 

-Student Activity Book " 
Cl«ar?y Action Transparency 
Action Sequence Cards 

FuncHons 

Functions Obfecfs Box 

L%es and Cbfects Corcfe " — 

"How Do We Use It" Cards ~ 

Audio Tape " ' 

Doing & Using Picture Cc^ds ~ ^ ^ 

Puzzle Pieces 

Touch & Feel!A/ioterials " * ^ 

Student Activity Book *~ " ~" 
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ClossificQtion 
Audio Tape 
Student Activity Bo<^ 
Picture and Game Cards 
Instructo Activity Kit 
Best Word Book Ever 
Total Score 



dumber of Objectives actually covered '\ _ 
\ Number of objectives in lesson / 



5« Teacher effectiveness in maintaining student attention and elicitation 
of shjdent responses. 

1 2 3 4 5 

6* Student interest toward the lesson, 

1 2 3 4 5 

7 Extensitv of covera e z' *^^^^'' objectives actually covered \ _ 

ensity coverage ^ Number of objectives in lesson / — 

8* Reported preparation time for lesson (in minutes) 
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Instnjcfions for Use of the LAT Level A 
Classroom Observation Form 



The observation fomi should be completed immediat^r^ly after observing 
the class. Nothing on the form is to be written during the class session or in 
the presence of the classroom teacher. 

If, at any time, a teacher should ask the purpose of the observation(s)y 
you are to indicate that you are observing the class to see how the students 
react to the materials i.e., if the students en{oy the lessons, like to partici^ 
pate, and to obtain teacher feedback concerning the usability of the LAT 
packages and materials. 

When observing a class, be sure you have a copy of the teachers guide 
currently being used. Determine which lesson the teacher is presenting and 
carefully follow the lesson content and procedures. 

Demographic data: 

These items are self-explanatory 

Item 1: Time on lesson 

Indicate the time (to the nearest minute) the lesson began (a), the 
time the lesson ended (b) and the total time b-a. If the teacher 
began the lesson before you had arrived, ask her (at the end of the 
lesson) when she had begun. 

Item 2: Teacher preparedness for LAT lessons 

Score: 

1 -° If the teacher has obviously made no preparation for the lesson, 

the materials are not readily available and the teacher must 
literally read from the manual throughout the lesson. 

2 - For situations between 1 and 3. ■ 
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3 - If the teacher is somewhat prepared for the lesson, has some 

of the mater lols at hand and occasionally refers to the teacher's 
guide. 

4 - For situations between 3 and 5. 

5 * if the teacher is obviously well prepared for the lesson, has all 

the materials at hand and can present the lesson smoothly with- 
out having to refer to the teacher's guide* 

Item 3. Correct following of procedures as specified in the teacher's guide. 

Scoring: 

For each of the LAT lessons, several procedures are specified in 
the Teachers Guide. The score for following procedures is com- 
puted by determining the actual number of the recommended pro- 
cedures that were Followed in the lesson and dividing the total 
number of procedures that were specified for. the lesson. For 
example, if seven procedures are listed for a given lesson and 
the teacher follows four of the procedures, the score would be 
4/> or .57. 

Item 4v Proper use of LAT materials 

Scoring: 

Place a check {V) by those materials that are supposed to be used 
during the particular lesson, indicate (by using 1 or 0) whether 
or not each of the suggested materials were or were not used. 
Also indicate (by using 1 or 0) whether or not each of the 
materials were or were not used according to the recommended 
procedures specified in the teacher's guide. The total score is 
computed as follows: 

(a) Compute the total possible score e.g.. If three materials 

were to be used in the lesson (according to the teacher's 
guide) the total possible score would be 6 (i.e., one 
* point each for usage and proper usage for 2 points per 
moteriaOo 

(b) Compute the actual score e.g., if two of the three materials 
were used and used properly, the actuol score would be 4; 

(c) The total score is 4/6 or .66. 



61 



item 5: Teacher effectivness :n maintaining student attention and 
elicitation of student responses. 

Score: 

1 - If the teacher does not have the attention of most of the 

students and there is almost no student participation or 
response. 

2 - If the teacher does not have the attention of the majority 

of the students and there is little student participation or 
response* 

3.- If the teacher has the attention of some of the students 
and con cltcit some student response and maintains some 
student attention throughout the lesson « 

4 - If the teocher has the attention of the majority of the 

students and maintains a fbrily high level of response 
and participation throughout the lesson. 

5 - If almost all of the students appear interested in the lessons, 

^njoy the activities and volunteer to participate. 

Item 6. Student interest towards the lesson 
Score: 

1 • If most of the students appear bored or do not pay attention to 

the lesson and activities. 

2 " For situations between 1 and 3. 

3 - If some of the students appear interested in the lessons, enjoy 

the activities and volunteer to participate. 

4 - For situations between 3 and 5. 

5 - If most of the students appear interested in the lessons, enjoy 

the activities and volunteer to participate. 

Ite^n 7: Extensity of coverage 
Scoring: 

Some lAT lessons are desinged to cover a single objective while 
other lessons may cover two or three objectives. The score for 
extensity of coverage is obtained by dividing the number of 
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objectives actually covered in the lesson by the number of 
objectives that should have been covered. For example, if 
the teocher should have covered three objectives (as indicated 
in the teacher's guide) and actually covered only one 
o^jjective, the score would be .33. 

Reported preparation time for lessons. 

After the teacher has completed the LAT lesson, ask her now much 
time she typically devotes to preparing for LAT lessons. 
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APPENDIX in 
K.E.D. PROGRAM TEACHER QUESTIONIMAIRE 
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K«E«D, Teacher Questionnaire 

1 • Nome: 



2. Location of KED class(s) AM School 

PM_ School 

3* How many years experience have you hod (to the nearest 1/2 year): 

a) Teaching kindergarten prior to the KED program) years 

b) teaching in the KED program years 

c) teaching in other grades including years listed under (a) years 

5. Do you have an aide for your 

a) AM KED Class yes _no 

b) PM KED Class yes no 

9. Please circle the letter preceding each phrase which explains why some 

children in your class(s) did not benefit from the KED program (if you think 
:sonie children did not benefit sufficiently from the program), 

immaturity 

b, physical handicaps (i,e«, poor vision or hearing) 
Co class size too large for individual attention 
d« content too advanced 

(specify which component ^LAT. _ ^DUSO BRL) 

e. content did not hold children's interest 

(specify which component L AT ^DUSO BRL 

fo short attention span 

some children transferred in too late to venefit from the program 
h. other - (pleose specify) 
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For each of the followIngT" circle the tenm which best reflects your 
reactions to the LAT teacher's guides. 



a. The lessons and content are: 

^^'eil organized adequately organisied 

b« The lessons are: 

too structured well balanced 

Cm The directions are: 

clearly written adequately written 

dm The lessons and content are: 

well sequenced reasonably sequenced 

e« The procedures are usually: 

easy to follow difficult to fellow 



nof well organized 
too flexible 
unclear or ambigous 
poorly sequenced 



In each row below, cif-cle the term vhich best reflects your reaction to 
the LAT components* 



a. Too many right amount 

b. The components are (very useful 

c. The components are (easy to handle 

d. The components are (durable flimsy) 

e. The component^ are (very interesting somewhat interesting 

not interesting) to the students. 



too few 

somewhat us^^^ul 
unnecessary) 

difficult to handle) 
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19. How helpful were the pracHce tests accompanying the LAT packages in 
preparing your students for the LAT Mastery Criterion Tests? 

a) very helpful 

b) somewhat helpnji No response 

c) unnecessary for my students 

20. Wos the LAT inservice adequate in preparing you to use the LAT materials? 

a) yes b) no If not, how could it be improved? 



23. As a result of the inservice, how confident were you in your ability to 
properly use each of the following KED curriculum components? 

LAT a) very confident b) confident c ) not very confident 

OUSO a) very confident b) confident c) not very confident 

BRL a) very confident b) confident c) not very confidant 
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